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Abstract: This article presents the methodology used by the Finnish Education Evaluation Council in the 
national evaluation of social and communication skills in vocational education and training. The evaluation 
concentrates on key competences such as leaming-to-learn skills, communication skills, social skills and 
entrepreneurship (Implementation of Education and Training 2010, 2004; Winterton, Delemare-Le Deist & 
Stringfellow, 2005; Recommendation of the European Parliament and of the Council of December 18, 2006 on 
key competencies for lifelong learning, 2006). The evaluation focuses particularly on skills required in the labour 
market. These competences are included in all vocational study programmes as a part of accompanying with other 
key competences. The authors’ approach follows the principles of developmental and participative evaluation, 
applying the Common Quality Assurance Eramework (CQAE) model, which has been accepted by the European 
Commission for the quality management and quality assurance framework for vocational education and training in 
Europe. Additional methods such as interview and statistical data gathering were used to acquire comprehensive 
data and ensure extensive participation of each partner involved has key competences. The evaluation material 
was collected from vocational education and training providers by sector (n=8) and by qualification (n=53). Every 
sector and type of examination was included in the evaluation. The evaluation was based on a sample (n=130). 
The evaluation data were collected from multiple sources by using multiple methods. The material included 
written self-evaluation reports by vocational education and training providers (n=130 which equals 95% of VET 
providers), quantitative information, interviews organized by the National Board of Education and an evaluation 
seminar for providers as focus groups. Educational administrators, teachers, students, employers and workplace 
instructors participated in the self-evaluation of the vocational education and training establishments 
(multi-professional evaluation). 

Key words: vocational education and training (VET); evaluation of education and training; evaluation; 
social skills; communication skills; key competences 

1. Introduction 

In recent years, the structure of vocational education and training in Einland has been reorganised, the 
amount of on-the-job learning has increased and skills demonstration has been introduced in all sectors and for all 
qualifications. The importance of general key competences as part of vocational competence has also been 
emphasized. These key competences will be further highlighted in the new national requirements for upper 
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secondary vocational qualifications now being prepared. Key competences in lifelong learning will be developed 
according to the recommendations given by the European Parliament and Council (Recommendation of the 
European Parliament and of the Council of December 18, 2006 on key competencies for lifelong learning, 2006). 

This evaluation was a national evaluation project commissioned by the Ministry of Education. The purpose 
of the evaluation was to analyse the teaching of social and communication skills for working life as part of the 
three-year vocational education and training programmes. As specified by the Ministry, the evaluation was to 
concentrate on the following skill areas: (1) social skills (e.g., ability to work with others and to cope with 
work-related social situations); (2) communication skills (e.g., presentation skills, language skills and customer 
service skills required in the job); (3) leaming-to-leam skills (e.g., learning capacity, self-improvement, 
self-evaluation of work performance); and (4) entrepreneurship skills and skills for functioning as an independent 
professional. 

The evaluation focused both on the activities designed to develop these skills and on the actual results of this 
by the end of the education and training. The evaluation considered how education and training providers and 
educational institutions have taken the need of these skills into account in their curricula, teaching, on-the-job 
learning, student assessments, teacher training and their overall activities as education and training providers, and 
the results achieved were also evaluated. The skills acquired through education and training were assessed in the 
light of learning results data from vocational skills demonstration. The national core curriculum and teacher 
training were also evaluated. 

2. Methodology of the evaluation 

The evaluation followed the principles of developmental, participative and collaborative evaluation (e.g., 
Patton, 1997; O’Sullivan, 2004) and applied the Common Quality Assurance Eramework (CQAE) model (i.e.. 
Vocational Education and Training). The data for the evaluation was collected from multiple sources by using 
multiple methods. The materials included self-evaluation reports written by providers of vocational education and 
training, statistical information, and interviews with teachers, education administrators and teacher education 
units. 

One of the greatest challenges was conceptually defining the key competences of lifelong learning. This 
involved referring to the literature on the subject, examining interpretations emerging in the drafting of curricula 
and qualification requirements, and practical experiences in working life. The concepts were pragmatically 
defined by a multi-professional team. The concept of entrepreneurship was divided into three sub-categories: 
“self-motivated entrepreneurship”, “internal entrepreneurship” and “external entrepreneurship”. 

The material was collected separately for each qualification to facilitate exploration of special features in and 
differences between sectors and qualifications. The evaluation involved all sectors and qualifications except for 
the “circus” sector (n=52). Two education and training providers were chosen to represent each qualification. The 
evaluation also covered different types of training (training for young people, adult education, apprenticeship 
training, both Einnish and Swedish speaking education and training and special needs education). The responses 
covered 127 qualifications (response rate: 93.4%). 

Qualification-specific working groups wrote descriptions and generated quantitative data, illustrating the 
state of the sector at the time of evaluation. After the collating of data and the writing of descriptions, the 
organizers held an evaluation session, with representatives of teachers, management, workplace instructors and 
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students participating. 

Good practices were compiled from the qualification descriptions written by the education and training 
providers. 

The external evaluation was performed by a multi-professional evaluation group (six members). At the first 
stage, the qualification-specific materials and best practices were analysed and subjected to criteria-based 
evaluation. The criteria were based on the CQAF model and target analysis. The main focus was on systematic 
operations (goals, operations, evaluation and monitoring, continuous improvement, plus results corresponding to 
goals). The interpretive analysis was carried out both numerically (on a scale from 0 to 5) and verbally. The 
highest possible point score was 140, and the score components were shown in Table 1 below: 



Table 1 Evaluation targets and points 



Needs of working life emphasized in education and training 


0-20 points 


Planning 


0-25 points 


Teaching and studying 


0-30 points 


Monitoring and assessment 


0-10 points 


Improvement and development 


0-15 points 


Results and effectiveness 


0-20 points 


Training policy and policy of the education or training provider 


0-20 points 



The evaluation also set a threshold value for high-quality operations. The evaluation group set the score of 
“3” as the threshold value: “some clear evidence that operations are consistent with the goals, both in planning 
and in practice, and some clear evidence that the results match the goals”. This was converted into an overall 
quality threshold: 57% of the maximum point score. On the basis of this, the education and training providers and 
qualifications were divided into two main groups: those exceeding the quality threshold and those falling short of 
the quality threshold. In addition to the scoring, each evaluator analyzed the quality of the results and the action, 
and wrote a report based on his or her own interpretive analysis. The national evaluation was compiled on these 
analyses, and the results were interpreted together during the evaluation sessions (n=9). 

At the second stage, further evaluation information was sought. Learning results data compiled by the 
National Board of Education was used, representatives of authorities and teacher training institutions were 
interviewed, and the revised national requirements for upper secondary vocational qualifications were analysed 
(using the vocational qualification in construction as an example). A survey was also circulated to teacher training 
institutions, and an evaluation seminar (focus group discussions) was held to education and training providers to 
analyse the evaluation results together with teachers. The material supplied by the education and training 
providers was also used to create a new “in-depth analysis” focusing on the integration of social and 
communication skills into other areas of teaching and studying (see Figure 1). The extent of this integration was 
judged using the following criteria: (1) integrated pedagogy (e.g., profound awareness of the key competences of 
lifelong learning; implementation shared between various teachers and workplaces; inclusion of social skills in 
student assessment); (2) semi-integrated pedagogy (e.g., responsibility for these competences rests with one 
teacher); and (3) non-integrated pedagogy (e.g., no-one consciously takes responsibility for the key competences). 

The external evaluation was organized in a multi-professional network by the Finnish Education Evaluation 
Council. It was carried out by an evaluation group, the members of which represented education and training 
providers, the employers’ national federation, experts of the Finnish National Board of Education and researchers 
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from various universities. 



Social and communicational skills 



steering system 

Needs of working life 
National goals 
Concepts; 

-Key competencies 
-Core competence 
-Common focus 
Norms; 

-Curriculum/require- 
ments for vocational 
qualifications 
-T eaching/on-the-j ob 
learning 

-Student assessment 
-Skill demonstration 
- Evaluation learning 
outcomes 



Prosess quality: How is it taken into account? 



Social skills 



1 Qualification-specific descriptions and 
evaluation panel 

2 Seminar: focus group discussions 

3 Interview with the authorities 



Learning to learn skillsX. , .4. . . 

houbleshooting skills > m-depth analysis 



Communication skills 



5 Requirements for upper secondary 
vocational qualifications (construction 
sector) 

> 

6 Surveys and interviews for teacher 
training institutions 



Entrepreneurship skills 




Analysis of evaluation area and earlier results 

Decision Dec.2006 Planning Apr. -Aug. 07 Material Nov.07 



Quality of 
results: What is learned? 



1 Descriptions and 
evaluation 

2 Learning outcomes 
- follow-up data 
(The Finnish National 
Board of Education) 



Evaluation Dec. 07-Sept. 08 Report Jan. 09 



L 



Development evaluation framework - the evaluation process 



Figure 1 The evaluation framework and process 



3. Results and discussion 

3.1 Education and training are advancing in line with the goals 

The evaluation results show that the general goals regarding social and communication skills have already 
been attained, and that vocational education and training is progressing according to these goals. However, 
development challenges still remain. 
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Figure 2 Distribution of scores for social and communication skills in upper secondary vocational qualifications 
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Figure 2 shows that the most qualifications are to be found at the average point score levels of 80 to 89 (n=25) 
and 60 to 69 (n=19). None of the qualifications reached the maximum point score (140), though on the other hand 
none of the qualifications received the minimum score either. 

(1) About half of the qualifications in the evaluation (50.4%) attained the quality threshold, set at 57% of the 
maximum, or 79.8 points out of 140 (Table 1); 

(2) 24% of the qualifications (n=30) passed the quality threshold in all sections and 45% (n=57) in some 
sections; 

(3) 32% of the qualifications did not pass the quality threshold in any evaluated section. 



Table 1 Quality level by evaluation area 



Evaluation area 


(a) Education or training provider 
falls short of quality threshold 


(b) Education or training provider 
exceeds quality threshold 




N 


% 


N 


% 


Needs of working life 


59 


46.5 


68 


53.5 


Planning 


76 


59.8 


51 


40.2 


Implementation of teaching and studying 


66 


52.0 


61 


48.0 


Monitoring and assessment 


64 


50.4 


63 


49.6 


Improvement and development 


73 


57.5 


54 


42.5 


Results and effectiveness 


61 


48.0 


66 


52.0 


Policy 


75 


59.1 


52 


40.9 


Total score 


63 


49.6 


64 


50.4 



3.2 Activities aimed at teaching social and communication skills are of fairly even quality, but generally 
they fail to reach the threshold level 

Figure 3 shows that the quahty of activities aimed at teaching social and communication skills is fairly even 
between the various areas evaluated, but they almost consistently fail to reach the threshold level of quality. 




Figure 3 Level of quality for the areas evaluated 
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The best results were observed in anticipating and adapting to the needs of working life and in activities 
related to results and effectiveness. The lowest scores were observed in planning, education and training provider 
policy, improvement and development. 

Sector- and qualification-specific differences were significant. In “technology and transport” and “natural 
resources and environmenf’, the point scores fell below the quality threshold in every area. Special educational 
institutions put a great deal of emphasis on social and communication skills, while apprenticeship training focuses 
least of all on these. The differences were distinct in different educational fields. It is a common idea that the 
social and communication skills in adult education and apprenticeship education are considered skills acquired 
and improved during the lifetime, and therefore they are not paid any attention to during the training. There were 
only negligible differences between Finnish-language and Swedish-language training. 

3.3 Interpretive analysis revealed some factors and development needs related to the quality of the 
action 

According to the interpretive analysis, many providers had set a goal for social and communicational skills 
and lined up activities connected to them. Many providers were lacking definitions of policy and orientation, in 
which case planning and implementation accumulated on individual teacher’s responsibility. The problem was, 
that providers’ policy did not always meet reality in educational institutions, and guidelines in an individual 
qualification did not come true. The distance between the educational or training institution and the education or 
training provider causes the problem of not seeing each other’s needs. It is a cultural question. 

Education providers and educational institutions are aware of working life demands, although their 
anticipation methods are fairly undeveloped. The most critical point in the planning was related to the aims which 
for many were unclear, and partly totally missing. A major problem was also, that the social and communicational 
skills were not specified but taken as a whole. This highlighted, among other things, the need to define the 
concepts. 

In pedagogical activity was emphasized both the significance of printed curricula and student assessment and 
vocational skills demonstrations. In educational institutions, very little attention was paid to the connection 
between printed and realized curriculum. The significance of the hidden curriculum turned out to be notable. 

The cooperation between different teacher groups is unusual. Confusions occurred in responsibilities in 
teaching social and communicational skills. In some qualifications, the teachers of vocational studies had this 
responsibility, in some others, it was on the teachers of the core subjects (mother tongue and foreign languages 
among others), whereas in some studies the whole responsibility was shifted to the on-the-job-training. Relatively 
common opinion seems to be, that the social and communicational skills develop naturally along with 
teaching/training and other communal activity as well as learning on the job, without special investments. 

The teachers orientated themselves to teach social and communicational skills through the teaching methods 
and the subject. Teaching supports group work and teamwork and emphasizes cooperational methods and 
customer service, on-the-job-leaming and student assessment. A closer look at teacher’s work brought out a 
problem of a single teacher: Vocational studies teachers teach also core studies, such as mother tongue, foreign 
languages and mathematics. 

Student assessment was a great pedagogical challenge. Social and communicational skills are evaluated at the 
workplace more often together with on-the-job-leaming and vocational skills demonstrations than with other 
student assessment. Usually there are no common assessment criteria but the assessment is teacher-specific. 
Assessment of social and communicational skills was more formative than summative. 



41 




Social and communicational skills in upper secondary vocational education and training 



Many providers and educational institutions had rather organized evaluation and follow-up systems, but only 
a few included in them the follow-up and evaluation connected to the social and communicational skills. In some 
cases, the evaluation and follow-up procedures were totally lacking. The most essential matter was the feedback 
from workplaces. The most critical point was the evaluation of the learning outcomes. In great part of the 
qualifications, learning outcomes are not at all evaluated, yet in spite of that, the teachers were able to present 
results concerning students’ knowledge (=insight). The problem in learning outcome evaluation was also, that the 
fields related to the social and communicational skills are not specified at all, but are understood as one unity. 
Besides the student assessment, different qualifications also needed improvement of the evaluation methods which 
would support teachers work (such as peer evaluation), and also deepening the culture of learning from others. 

There was great variation between qualifications in quality improvement and developing. Some 
qualifications had separate development projects related to social and communicational skills, and some had 
connected social and communicational developing potentials to be a part of general improvement. Only little 
evidence of pedagogical innovations could be seen in the data. The problem here was, that development 
operations are not seen as a vision but as actions, such as development projects. The future development trends 
were quite narrow and technically oriented. 

Although economical and other resources were estimated rather good, more resources are needed for the 
cooperation between different teacher groups and concerted teaching, for the training of teachers and workplace 
instructors, for integrating teaching in different special fields, and especially for developing student assessment. 
Pedagogical leadership (process management) was brought out to be a central development object. 

Social skills were emphasized more than other special fields. Language teaching was especially seen a 
support in building up communicational skills. In practice, leaming-to-leam skills were most often entrusted to 
individual teachers and students. Entrepreneurship was considered important, though students of vocational upper 
secondary education and training were regarded too young and inexperienced to study entrepreneurship. Some 
thought that education provides good basic skills for entrepreneurship. 

Several critical comments were submitted to national steering and development interests were brought out. 
Accuracy in national guidelines and objectives, support in developing new methods and culture and teacher 
training were especially wanted. 

3.4 Considerable differences in the extent of integration of social and communication skills into other 
areas of teaching that is of great importance for the development of these skills 

Teaching of these skills was found to be integrated in less than 30% of the qualifications, semi-integrated in 
less than 40% and non-integrated in more than 30%. Integration was best achieved in the “humanities”, 
“education” and “science” fields, and least well achieved in the “technology and transport”, “natural resources and 
the environmenf’, and “tourism, catering and domestic services” fields. The largest number of qualifications with 
no integration was found in the “natural resources and the environmenf’ field. Integration was better achieved in 
special needs teaching than in other forms of education, and the lowest integration rate was found in 
apprenticeship training. 

3.5 The need for social and communication skills in working life is greater than what the education and 
training system can cater for 

Social and communication skills are highly important in working life, but in education and training they are 
not given as much weight as they deserve. Leaming-to-learn and social skills receive the most focus, while the 
least focus is given to communication skills and entrepreneurship. The results, i.e. the level achieved through 
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education and training, are estimated to be some 20 percentage points lower than the needs of working life. The 
entrepreneurship results fall below the quality threshold. 

The greatest gaps between the needs of working life, the focus in education and training, and the results 
achieved were found in qualifications where the pedagogy approach was non-integrated. By contrast, greater 
focus is placed on the needs of working life in qualification groups with an integrated pedagogy approach (see 
Figure 4). 




social skills 

— ^ communication 
skills 

learning-to-learn 
— ^ entrepreneurship 
qual ity threshold 



Figure 4 Need for social and communication skills in working life, the emphasis given to them in education and training, 

and how their point scores relate to the quality threshold 



3.6 The learning results are good 

In educational institutions, the social and communicational skills of the students are considered to be very 
good. Easily getting a job, small amount of dropouts and positive feedback from employers were also regarded 
good indicators of social and communicational skills. 

The National Board of Education monitors and evaluates the learning results of vocational education and 
training (Rakkolainen, M., 2005). The evaluation data is generated directly from vocational competence tests, 
without national tests, and evaluation data is also generated on lifelong learning skills. The evaluation shows too 
good learning results: The most common point score in lifelong learning skills is 4 (out of a maximum of 5). 
There are great differences from one qualification to another in terms of the mastery of key competences. Women 
generally score better in these skills than men. 

3.7 Teacher training is not very systematic in preparing teachers to take social and communication 
skills into account in vocational education and training 

Teacher training institutions vary greatly in their awareness of the growing need for social and 
communicational skills teaching in vocational education and training. There are also widely differing views on the 
needs of working life. Procedures for anticipating those needs vary from one institution to another, and differences 
between sectors were considered great. 

There is much variation in the extent to which the needs of working life and vocational education and 
training institutions are taken into account in teacher training curricula. There is also variance in the extent to 
which teachers are instructed in how to take social and communicational skills into account in their future jobs. In 
continuing education for teachers, the focus on social and communicational skills also differs greatly from one 
teacher training institution to another. 

Educational institutions and education providers adopted a very critical attitude towards teacher training in 
their self-evaluation. The sharpest criticism was focused on the scientific training methods and abstracting the 
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training from the needs of daily work and life of young people. 

3.8 The new national requirements for upper secondary vocational qualifications do not sufficiently 
clarify the inclusion of social and communicational skills in vocational education and training 

In the new national requirements for upper secondary vocational qualifications, great changes have been 
made to the structure and content compared with the earlier national core curriculum and requirements for 
qualifications regarding social and communicational skills (Basic qualification in building trade, 2008). The 
harmonization of concepts achieved with this reform will clarify the vocational qualification system, for instance 
with regard to its relationship to the needs of working life. 

However, key competences in lifelong learning have not been unambiguously defined in the new framework, 
and it may therefore turn out to be difficult to identify the essential features of each of those competences. The 
anticipated threats are that key competences in lifelong learning: (1) are understood as one package and not 
addressed separately; (2) will continue to be left for a single teacher to handle; and (3) are too complicated for 
people in working life to evaluate. 

4. Conclusion 

The evaluation demonstrated that vocational education and training is in fact progressing in line with the 
goals of lifelong learning and the needs of working life. Social and communication skills form part of vocational 
education and training, and greater importance is being attached to them in revisions to the framework for 
qualifications. Vocational education has become and will continue to become increasingly dependent on the needs 
of working life, through on-the-job learning and competence tests. Learning environments have become 
increasingly oriented towards working life. 

The evaluation raises implications how the key competences affect learning methods and outcomes. It also 
shows a tension between espoused aims and plans to cover these competences and the reality of teaching and 
assessment practices. In this paper we argue that teachers need better integration of aims and practices as well as 
clearer pedagogical strategy in order to take responsibility for training in all different skills. 

The major challenges concern the great differences observed between education and training providers, 
sectors and qualifications. There are differences related to operating practices and occupational traditions. The 
challenges are particularly apparent in the technology, transport, natural resources and environment sectors. The 
depth of integration in the pedagogical approach is an important challenge in changing operating practices. A 
further issue is that the special features of social and communication skills are not analysed and itemized; instead, 
these skills are often considered to be one package. Clarifying the concept of entrepreneurship is the greatest 
development challenge in this area. 

The evaluation group also presented improvement proposals in its report. These refer to education and 
training providers, qualifications, teacher training, national policies and official supervision. 
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